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- The MAILING DATE of this communication appears on the cover sheet with the correspondence address- 
All claims being allowable, PROSECUTION ON THE MERITS IS (OR REMAINS) CLOSED in this application. If not included 
herewith (or previously mailed), a Notice of Allowance (PTOL-85) or other appropriate communication will be mailed in due course. THIS 
NOTICE OF ALLOWABILITY IS NOT A GRANT OF PATENT RIGHTS. This application is subject to withdrawal from issue at the initiative 
of the Office or upon petition by the applicant. See 37 CFR 1.313 and MPEP 1308. 

1 . This communication is responsive to RCE filed on 01/09/08 . 

2. £3 The allowed claim(s) is/are 1-20 . 

3. □ Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 

a) O All b)DSome* c) □ None of the:' 

1 • □ Certified copies of the priority documents have been received. 

2. □ Certified copies of the priority documents have been received in Application No. . 

3. □ Copies of the certified copies of the priority documents have been received in this national stage application from the 

International Bureau (PCT Rule 17.2(a)). 
* Certified copies not received: . 

Applicant has THREE MONTHS FROM THE "MAILING DATE" of this communication to file a reply complying with the requirements 
noted below. Failure to timely comply will result in ABANDONMENT of this application. 
THIS THREE-MONTH PERIOD IS NOT EXTENDABLE. 

4. □ A SUBSTITUTE OATH OR DECLARATION must be submitted. Note the attached EXAMINER'S AMENDMENT or NOTICE OF 

INFORMAL PATENT APPLICATION (PTO-152) which gives reason(s) why the oath or declaration is deficient. 

5. □ CORRECTED DRAWINGS ( as "replacement sheets") must be submitted. 

(a) □ including changes required by the Notice of Draftsperson's Patent Drawing Review ( PTO-948) attached 

1) □ hereto or 2) □ to Paper No./Mail Date . 

(b) □ including changes required by the attached Examiner's Amendment / Comment or in the Office action of 

Paper No./Mail Date . 

Identifying indicia such as the application number (see 37 CFR 1.84(c)) should be written on the drawings in the front (not the back) of 
each sheet. Replacement sheet(s) should be labeled as such in the header according to 37 CFR 1.121(d). 

6/ □ DEPOSIT OF and/or INFORMATION about the deposit of BIOLOGICAL MATERIAL must be submitted. Note the 
attached Examiner's comment regarding REQUIREMENT FOR THE DEPOSIT OF BIOLOGICAL MATERIAL. 
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Continued Examination Under 37 CFR 1.114 

1. A request for continued examination under 37 CFR 1.114, including the fee set 
forth in 37 CFR 1.17(e), was filed in this application after allowance or after an Office 
action under Ex Parte Quayle, 25 USPQ 74, 453 O.G. 213 (Comm'r Pat. 1935). Since 
this application is eligible for continued examination under 37 CFR 1.114, and the fee 
set forth in 37 CFR 1.17(e) has been timely paid, prosecution in this application has 
been reopened pursuant to 37 CFR 1.114. Applicant's submission filed on 01/09/08 has 
been entered. 

Information Disclosure Statement 

2. The information disclosure statement (IDS) submitted on 01/09/08 was filed after 
the mailing date of the Notice of Allowability on 10/16/07. The submission is in 
compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure 
statement is being considered by the examiner. 
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EXAMINER'S AMENDMENT 



3. An examiner's amendment to the record appears below. Should the changes 
and/or additions be unacceptable to applicant, an amendment may be filed as provided 
by 37 CFR 1.312. To ensure consideration of such an amendment, it MUST be 
submitted no later than the payment of the issue fee. 



Authorization for this examiner's amendment was given in a telephone interview with 
applicant's representative Mr. Jen C. Jenkins on 09/25/07. 



4. The application has been amended by replacing claims 1,10 and 11 with 
the following: 

1. A method of detecting a three-dimensional object adjacent to a user- 
interactive side of a surface that optically diffuses light, as a function of an infrared light 
transmitted toward the three-dimensional object from an opposite side of the surface 
and reflected back through the surface from the three-dimensional object to be received 
by a light sensor disposed on the opposite side of the surface, comprising the steps of: 

(a) • creating a first pixilated image representing the intensity of the 
infrared light reflected from the three-dimensional object and received by the light 
sensor; 

(b) creating a first binarized image from the first pixilated image by 
filtering out pixels of the first pixilated image that do not have an intensity exceeding a 
first threshold value, the first binarized image representing a first planar distance of the 
three-dimensional object from the interactive side of the surface; 

(c) creating a second binarized image from the first pixilated image by 
filtering out pixels of the first pixilated image that do not have an intensity exceeding a 
second threshold value, the second binarized image having substantially equal area 
and coordinate locations as the first binarized image, the second binarized image 
representing a second planar distance of the three-dimensional object from the 
interactive side of the surface; 
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(d) detecting a first connected component in the first binarized image, 
the first connected component representing a first set of pixels that have an intensity 
exceeding the first threshold value, the first connected component having a first 
bounding area defined by a first outer boundary and within which the first set of pixels 
reside and within which no pixels of the first binarized image have an intensity that fails 
to exceed the first threshold value; 

(e) determining the first bounding area and a first coordinate location of 
the first connected component in the first binarized image; 

(f) detecting a second connected component in the second binarized 
image, the second connected component representing a second set of pixels that have 
an intensity exceeding the second threshold value, the second connected component 
having a second bounding area defined by a second outer boundary and within which 
the second set of pixels reside and within which no pixels of the second binarized image 
have an intensity that fails to exceed the second threshold value; 

(g) determining the second bounding area and a second coordinate 
location of the second connected component in the second binarized image that has 
substantially equal area and coordinate locations as the first binarized image; and 

(h) determining that both the first connected component and the 
second connected component correspond to the three-dimensional object adjacent to 
the interactive side of the surface, if one of the first connected component and the 
second connected component is disposed substantially within the bounding area of the 
other of the first connected component and the second connected component. 



10. A computer-readable medium encoded with computer-executable 
instructions for carrying out the steps of Claim 1.' 



11. A system for detecting a relative position of a three-dimensional object, 
the system comprising: 

(a) a surface that diffuses light and has: 

(i) an interactive side adjacent to which the three-dimensional 
object can be manipulated; and 

(ii) an opposite side that is opposite the interactive side; 

(b) a light source spaced away the opposite side of the surface, the 
light source emitting an infrared light that is transmitted through the surface to the 
interactive side of the surface; 

(c) a light sensor disposed on the opposite side of the surface so as to 
sense infrared light reflected back from the three-dimensional object, through the 
surface; 

(d) a processor in communication with the light sensor; and 
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(e) a memory in communication with the processor, the memory 
storing data and machine instructions that when executed by the processor cause a 
plurality of functions to be carried out, including: 

• (i) creating a first pixilated image representing the intensity of 
the infrared light reflected from the three-dimensional object and received by the light 
sensor; 

(ii) creating a first binarized image from the first pixilated image 
by filtering out pixels of the first pixilated image that do not have an intensity exceeding 
a first threshold value, the first binarized image representing a first planar distance of 
the three-dimensional object from the interactive side of the surface; 

(iii) creating a second binarized image from the first pixilated 
image by filtering out pixels of the first pixilated image that do not have an intensity 
exceeding a second threshold value, the second binarized image having substantially 
equal area and coordinate locations as the first binarized image, the second binarized 
image representing a second planar distance of the three-dimensional object from the 
interactive side of the surface; 

(iv) detecting a first connected component in the first binarized 
image, the first connected component representing a first set of pixels that have an 
intensity exceeding the first threshold value, the first connected component having a 
first bounding area defined by a first outer boundary and within which the first set of 
pixels reside and within which no pixels of the first binarized image have an intensity 
that fails to exceed the first threshold value; 

(v) determining the first bounding area and a first coordinate 
location of the first connected component in the first binarized image; 

(vi) detecting a second connected component in the second 
binarized image, the second connected component representing a second set of pixels 
that have an intensity exceeding the second threshold value, the second connected 
component having a second bounding area defined by a second outer boundary and 
within which the second set of pixels reside and within which no pixels of the second 
binarized image have an intensity that fails to exceed the second threshold value; 

(vii) determining the second bounding area and a second 
coordinate location of the second connected component in the second binarized image 
that has substantially equal area and coordinate locations as the first binarized image; 
and 

(viii) determining that both the first connected component and the 
second connected component correspond to the three-dimensional object adjacent to 
the interactive side of the surface, if one of the first connected component and the 
second connected components is disposed substantially within the bounding area of the 
other of the first connected component and the second connected component. 
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Allowable subject matter 

5. Claims 1-20 as amended are allowed. 

6. The following is an examiner's statement of reasons for allowance: 

Regarding claiml , and similarly claims 1 0 and 1 1 , closest art of record do not 
disclose or fairly suggest all limitations. Specifically, 

• Fitzmaurice (US 2004/01 35824) discloses a tablet PC that detects touching of 
the pen 

• Speeter (US 5,479,528) discloses applying thresholding and CCL to locate 
fingertips on an intelligent work surface 

• Nettles et al. (US 5,424,823) discloses applying connected component labeling 
(CCL) to range images to detect vertical surfaces at different depth 

• Ma et al. (US 6,674,900) discloses applying multiple thresholding to generate 
multiple images and then applying CCL to each of the images 

• Gokturk et al. (US 2003/0235341) discloses using depth thresholding to segment 
foreground and background 

• Xu et al. ("A Robust Close-range photogrammetric system for industrial 
metrology," 7 th Int'l Conf. on Control, Automation, Robotics and Vision, Dec.2002, 
pp. 1 14-119) discloses multiple thresholding to obtain sub-pixel object location 
for stereo matching 
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• Tanaka et al. ("Development of a video-rate range finder using dynamic threshold . 
method for characteristic point detection," Int'l Conf. on Advanced Intelligent 
Mechatronics, 19-23 September 1999, pp. 932-937) discloses using dynamic 
thresholding to obtain more reliable boundaries 

• Rekimoto et al. ("Perceptual Surfaces: Towards.a Human and Object Sensitive 
Interactive Display," Proceedings of Workshop on Perceptual User Interfaces, 
1997, 3 pages. Submitted as part of the IDS) discloses using a predetermined 
threshold to binarize an image, following by CCL to detect hands 

However, none of the references cited above, alone or in combination, disclose or 
fairly suggest, among other things, the determination criterion recited in element (h) of 
claim 1 . 

7. Any comments considered necessary by applicant must be submitted no later 
than the payment of the issue fee and, to avoid processing delays, should preferably 
accompany the issue fee. Such submissions should be clearly labeled "Comments on 
Statement of Reasons for Allowance." 
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Contact Information 

8. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Yubin Hung whose telephone number is (571) 272- 
7451 . The examiner can normally be reached on 7:30 - 4:00. If attempts to reach the 
examiner by telephone are unsuccessful, the examiner's supervisor, Matthew C. Bella 
can be reached on (571) 272-7778. The fax phone number for the organization where 
this application or proceeding is assigned is 571-273-8300. 

9. Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 

Yubin Hung 
Patent Examiner 
Art Unit 2624 

February 18, 2008 




